The information complexity of learning tasks, their structure and their distance

Abstract We introduce an asymmetric distance in the space of learning tasks and a framework to compute their complexity. These concepts are foundational for practice transfer learning, whereby parametric model is pre-trained task, then fine tuned another. The we develop non-asymptotic, captures finite nature training dataset allows distinguishing from memorization. It encompasses, as special cases, classical notions Kolmogorov complexity Shannon Fisher information. However, unlike some those frameworks, it can be applied large-scale models real-world datasets. Our first measure way that accounts effect optimization scheme, which critical deep learning.